Learning About Bias?
An Experiment on Paying Attention to News in Russia

Anton Shirikov
as6859@columbia.edu

Columbia University

Georgiy Syunyaev
g.syunyaev@vanderbilt.edu

Vanderbilt University, WZB Berlin

The question


  • Can we prompt citizens to pay attention to media coverage?

  • …Does this lead to learning about media bias and switching to less biased media?

What we know


Why is re-evaluating the bias difficult?



Our argument


  • Citizens in autocracies can revise their beliefs about media bias if they pay closer attention to:

    • What pro-government media covers
    • Differences between what pro-government and independent media cover

  • But they need to make conclusions on their own to:

    • Prevent relying on others’ judgment
    • Avoid attributing evidence to researchers’ (or other actors’) political agenda

Intervention and outcomes

We ask citizens to analyze the topics and sentiment of video news headlines by state and independent media

…to answer…

Does paying attention to what media outlet covers and how affect:

  1. Knowledge of media outlets’ reporting strategies?

  2. Perceptions of media outlets’ credibility?

  3. Consumption of state and independent media and awareness of pro-government and critical news?

  4. Political attitudes?

Setting


  • Russia: an autocratic regime with state control of media environment

    • State media are heavily pro-government
    • But more critical media also available
    • The War in Ukraine exacerbated state media control \(\Rightarrow\) Hard test of our theory

Research design



  • Online panel experiment among a convenience sample of adult Russian citizens with TGM Research

  • Simple random assignment to one of three experimental groups with pro-government or independent media content tasks

  • 4 waves with experimental tasks are included after the survey in first 3 waves

    • At least 2 weeks between experimental tasks and outcome measurement
  • Today, we focus on the endline (wave 4) outcomes

Timeline


Treatments


  • Short video segments (news headlines) from three TV stations:

    • Kultura, a non-political TV channel that covers art, theater, architecture, etc.

    • Rossiya-1, one of the two main state-controlled propaganda channels

    • RTVI, a privately owned and editorially (semi-)independent TV station

  • Each news headline segment was:

    • Brief description of main events of the day (~2 minutes)
    • Shown in the beginning of evening prime-time news broadcasts
    • Focuses on informing about main events (selection) rather than carefully framing them (slant)

Experimental conditions

Evaluate 4-6 video segments per wave:

  • Placebo: News headlines ONLY from the non-political Kultura TV channel

  • Pro-government: News headlines from Rossiya-1 AND Kultura TV channels

    • Order: Rossiya-1 (Date 1) \(\Rightarrow\) Kultura (Date 1) \(\Rightarrow\) Rossiya-1 (Date 2) \(\Rightarrow\) Kultura (Date 2) …

  • Pro-government vs. Independent: News headlines from Rossiya-1 AND RTVI

    • Order: Rossiya-1 (Date 1) \(\Rightarrow\) RTVI (Date 1) \(\Rightarrow\) Rossiya-1 (Date 2) \(\Rightarrow\) RTVI (Date 2) …

Topic count questions



  • Which TV station did broadcast this video? Did the video mention [TOPIC]?

  • How many times the following topics were mentioned in the video:

    • Situation in Russia in a [negative/positive] tone?
    • Situation in other countries in a [negative/positive] tone?
    • Russian federal officials in a [negative/positive] tone?

Hypotheses

After treatment tasks, citizens will…

  • H1: Learn about the types of coverage of Rossiya-1 and RTVI
  • H2: Update perceptions of media: Rossiya-1 bias \(\Uparrow\), RTVI bias \(\Downarrow\), attributing news to Rossiya-1 and RTVI \(\Uparrow\)
  • H3: Adjust media consumption: Rossiya-1 \(\Downarrow\), RTVI \(\Uparrow\), preference for RTVI \(\Uparrow\); news awareness
  • H4: Update political attitudes

H1: Learning about coverage

  • Citizens recognize that Rossiya-1 covers
    • Foreign countries more and more negatively and
    • Federal authorities more and more positively

H1: Learning about coverage

  • Citizens recognize that RTVI covers
    • Foreign countries more and in a more positive tone and
    • Russia/federal authorities less and in a less positive tone

H2: Perceptions of news outlets


  • How likely is Rossiya-1/RTVI to cover Russia positively and foreign countries negatively?
  • Drawing attention to content encourages learning about reporting strategy
  • Control group means: uncertainty about RTVI coverage strategy but little uncertainty about Rossiya-1


Rossiya-1 coverage:
RTVI coverage:
Share correctly attributed news:
Domestic positive Foreign negative Domestic positive Foreign negative RTVI Rossiya-1
Rossiya-1+Kultura 0.023** 0.038** 0.039* 0.011 0.038 0.043***
[0.014] [0.017] [0.023] [0.022] [0.023] [0.017]
Rossiya-1+RTVI 0.019* 0.023* 0.066*** 0.087*** 0.093*** 0.022
[0.014] [0.017] [0.023] [0.021] [0.023] [0.018]
Hypotheses upr/upr upr/upr two/two two/two two/upr upr/two
Control mean 0.908 0.843 0.473 0.618 0.575 0.762
Observations 1170 1173 1171 1171 1176 1176

H2: Perceptions of news outlets


  • No learning about the bias of pro-government Rossiya-1
  • But improved perceptions of independent RTVI


Rossiya-1 is unbiased/independent RTVI is unbiased/independent
Rossiya-1+Kultura 0.001 -0.021
[0.020] [0.016]
Rossiya-1+RTVI -0.007 0.025**
[0.020] [0.015]
Hypotheses lwr/lwr two/upr
Control mean 0.433 0.561
Observations 1164 1163

H3: Media consumption


  • Questions about media usage last week
  • Positive shift in self-reported consumption of both Rossiya-1 and RTVI
  • Stronger revealed preference for RTVI in both treatment groups


Used last week:
Behavior:
Rossiya-1 How many state outlets RTVI How many independent outlets Prefer Rossiya-1 over RTVI
Rossiya-1+Kultura 0.065 -0.017* -0.018 -0.006 -0.044**
[0.030] [0.013] [0.019] [0.011] [0.024]
Rossiya-1+RTVI 0.048 -0.011 0.122*** 0.011 -0.097***
[0.031] [0.013] [0.025] [0.011] [0.026]
Hypotheses lwr/lwr lwr/lwr two/upr two/upr lwr/lwr
Control mean 0.720 0.247 0.086 0.069 0.895
Observations 1175 1175 1175 1175 1176

H3: News awareness


  • Familiarity from last week’s headlines from state and independent media
  • Increase in awareness of both pro-government and critical news


Awareness of news stories
Pro-government Critical
Rossiya-1+Kultura 0.043** 0.042**
[0.019] [0.017]
Rossiya-1+RTVI 0.034* 0.026*
[0.018] [0.017]
Hypotheses two/two two/upr
Control mean 0.340 0.215
Observations 1175 1175

H4: Political beliefs


  • Three indexes: (1) approval of various Russian authorities; (2) quality of government-provided services; (3) concern about Russia’s war on Ukraine
  • Lower support for the government and stronger concerns about Ukraine


Authorities approval Quality of public services Ukraine concern
Rossiya-1+Kultura -0.141** -0.071 0.089**
[0.067] [0.060] [0.047]
Rossiya-1+RTVI -0.101* -0.121** 0.058
[0.065] [0.060] [0.045]
Hypotheses lwr/lwr lwr/lwr upr/upr
Control mean -0.001 0.001 -0.002
Observations 1164 1164 1165

Strict test of the hypotheses


  • H1: Differences in coverage
    • Rossiya-1 vs. Kultura: More positive domestic coverage / More negative foreign coverage and positive coverage of authorities
    • RTVI vs. Rossiya-1: Less positive domestic coverage and coverage of authorities / Less negative foreign coverage
  • H2: Effects on media perceptions
    • Pro-government news only: Rossiya-1 covers domestic news positively \(\Uparrow\) / Rossiya-1 covers foreign news negatively \(\Uparrow\)
    • Pro-government vs. Independent news: RTVI covers domestic news positively \(\Downarrow\) / RTVI covers foreign news negatively \(\Downarrow\) / RTVI bias perceptions \(\Downarrow\)
    • Both treatments: Rossiya-1 bias perceptions unchanged / Correct attribution of respective channel’s news \(\Uparrow\)
  • H3: Effects on media consumption
    • Pro-government vs. Independent news: Self-reported RTVI consumption \(\Uparrow\)
    • Both treatments: Self-reported Rossiya-1 consumption unchanged / Recall of pro-government news \(\Uparrow\) / Recall of critical news \(\Uparrow\) / Choose to watch RTVI \(\Uparrow\)
  • H4: Effects on political attitudes
    • Both treatments: Approval of authorities \(\Downarrow\) / Evaluation of government policies \(\Downarrow\) / Concerns about the war in Ukraine \(\Uparrow\)

Hypothesis supported or unsupported

Mechanisms


  • Respondents who tried RTVI were mostly state media consumers
  • But perceptions of RTVI as unbiased and preference for RTVI only improved among Putin critics, not Putin supporters; existing Rossiya-1 consumers also not affected
  • News awareness: pro-gov’t news \(\Uparrow\) mostly among supporters, critical news \(\Uparrow\) mostly among critics

Interpretation:: Learning about coverage prompts updating about media bias or news preference when citizens are already open to critical information; tasks increased attention to news

Mechanisms


Effects on political beliefs:

  • Approval of authorities decreased and concern about Ukraine increased among critics and consumers of independent media
  • No effects among Putin supporters or Rossiya-1/state media consumers

Interpretation: Increasing attention to news can shift beliefs if news are consistent with prior dispositions/media diets; otherwise, little updating

Concerns over the design?


  • No pure control? No systematic evidence of placebo group changes over time

  • Political news push respondents away? No differences in patterns of attrition across treatment groups

  • Less attention to political news? High and similar rates of correct answers about the topic and channel across groups

  • Experimenter demand? No differences across treatment groups in respondents’ assessments of study goals

Takeaways


  • Learning about the slant of propaganda is possible when citizens are more attentive

    • But it does not make them more suspicious about propaganda

  • Exposure to new (critical) information sources and increased attention to media coverage can shift news interest away from propaganda and prompt belief updating

    • But mostly if citizens are at least somewhat skeptical

Next steps



  • Clarify the mechanism/causal path

  • Explore heterogeneous effects further

  • Trace changes in consumption/news awareness over time (not pre-registered)

  • Additional outcomes (beliefs about media environment, etc.)

References

Arceneaux, Kevin, Martin Johnson, and Chad Murphy. 2012. “Polarized Political Communication, Oppositional Media Hostility, and Selective Exposure.” The Journal of Politics 74 (1): 174–86.
Badrinathan, Sumitra. 2021. “Educative Interventions to Combat Misinformation: Evidence from a Field Experiment in India.” American Political Science Review 115 (4): 1325–41. https://doi.org/10.1017/s0003055421000459.
Broockman, David, and Joshua Kalla. 2022. “The Manifold Effects of Partisan Media on Viewers’ Beliefs and Attitudes: A Field Experiment with Fox News Viewers.” OSF Preprints 1.
Chopra, Felix, Ingar Haaland, and Christopher Roth. 2022. “Do People Demand Fact-Checked News? Evidence from u.s. Democrats.” Journal of Public Economics 205: 104549. https://doi.org/10.1016/j.jpubeco.2021.104549.
———. 2023. “The Demand for News: Accuracy Concerns Versus Belief Confirmation Motives.”
Feldman, Lauren. 2011. “Partisan Differences in Opinionated News Perceptions: A Test of the Hostile Media Effect.” Political Behavior 33: 407–32.
Garcia-Arenas, Javier. 2016. “The Impact of Free Media on Regime Change: Evidence from Russia.” http://www.eief.it/files/2016/01/02-jmp_garcia-arenas.pdf.
Garrett, R. Kelly, and Brian E. Weeks. 2017. “Epistemic Beliefs’ Role in Promoting Misperceptions and Conspiracist Ideation.” PLoS ONE 12 (9).
Gentzkow, Matthew, and Jesse M. Shapiro. 2006. “Media Bias and Reputation.” Journal of Political Economy 114 (2): 280–316.
———. 2010. “What Drives Media Slant? Evidence from u.s. Daily Newspapers.” Econometrica 78 (1): 35–71. https://doi.org/10.3982/ECTA7195.
Gläßel, Christian, and Katrin Paula. 2020. “Sometimes Less Is More: Censorship, News Falsification, and Disapproval in 1989 East Germany.” American Journal of Political Science 64 (3): 682–98. https://doi.org/10.1111/ajps.12501.
Guess, Andrew, Michael Lerner, Benjamin Lyons, Jacob M. Montgomery, Brendan Nyhan, Jason Reifler, and Neelanjan Sircar. 2020. “A Digital Media Literacy Intervention Increases Discernment Between Mainstream and False News in the United States and India.” Proceedings of the National Academy of Sciences 117 (27): 15536–45. https://doi.org/10.1073/pnas.1920498117.
Nyhan, Brendan, and Jason Reifler. 2010. “When Corrections Fail: The Persistence of Political Misperceptions.” Political Behavior 32 (2): 303–30.
Searles, Kathleen, Joshua P. Darr, Mingxiao Sui, Nathan Kalmoe, Raymond Pingree, and Brian Watson. 2022. “Partisan Media Effects Beyond One-Shot Experimental Designs.” Political Science Research and Methods 10 (1): 206–14. https://doi.org/10.1017/psrm.2021.21.
Shirikov, Anton. 2024. “Rethinking Propaganda: How State Media Build Trust Through Belief Affirmation.” The Journal of Politics. https://doi.org/https://doi.org/10.1086/729941.
Stein, Elizabeth A. 2013. “The Unraveling of Support for Authoritarianism: The Dynamic Relationship of Media, Elites, and Public Opinion in Brazil, 1972–82.” The International Journal of Press/Politics 18 (1): 85–107.
Stroud, Natalie Jomini. 2010. “Polarization and Partisan Selective Exposure.” Journal of Communication 60 (3): 556–76. https://doi.org/10.1111/j.1460-2466.2010.01497.x.
Syunyaev, Georgiy. 2021. “Limits of Independent Media: Experimental Evidence on Local Coverage of Healthcare in Russia.”
Taber, Charles S., and Milton Lodge. 2006. “Motivated Skepticism in the Evaluation of Political Beliefs.” American Journal of Political Science 50 (3): 755–69.